Control device and robot system

ABSTRACT

A control device comprising a processor configured to receive information on a captured image from an imaging device capturing an image from an operator, control a robot including a robot arm on which a stamp that forms a marker on an object and an end effector that performs work on a work target are allowed to be provided by being replaced, perform correlation between a robot coordinate system that is a coordinate system relating to the robot and an image coordinate system that is a coordinate system relating to the captured image, and perform the correlation based on a plurality of coordinates of a predetermined portion of the robot arm in the robot coordinate system and a plurality of coordinates of the plurality of markers in the image coordinate system when the plurality of markers are formed on the object by the stamp.

BACKGROUND 1. Technical Field

The present invention relates to a control device and a robot system.

2. Related Art

In the related art, there is known a robot system that includes a robotincluding a robot arm for performing work on a work target placed on awork stand and a fixed camera fixed so as to be able to image the worktarget. In such a robot system, the robot can perform various types ofwork in the real space based on an image captured by the fixed camera.For that purpose, it is necessary to calibrate (correlate) an imagecoordinate system of the image captured by the fixed camera and a robotcoordinate system as a reference of robot control.

As a calibration method, various methods have been proposed in therelated art. The most common method is a method in which processing oftouching up a marker disposed on a work stand with a robot to acquireposition information in the robot coordinate system and processing ofdetecting a position of the marker with a fixed camera to acquireposition information of the marker in the image coordinate system areperformed and calibration between the fixed camera and the robot isperformed by combining these two pieces of position information.However, in this method, there is a problem that it is necessary totouch up the marker with the robot, it takes time, and it is difficultto improve accuracy.

Accordingly, as a method for solving this problem, there has beenproposed a method of forming a marker at any place by a marking deviceusing a robot including a robot arm that includes the marking devicetogether with a hand for performing work on the work target (See, forexample, JP-A-2010-64200). With this, it is possible to omit work oftouching up a fixed marker and it is possible to shorten the calibrationtime.

However, the robot used in such a method includes a hand and a markingdevice at the tip end portion of the robot arm, the tip end portion isvery large, and weight thereof is heavy. For that reason, in such amethod, there is a problem that it is difficult to improve accuracy ofthe calibration. There is a problem that the hand becomes an obstacle atthe time of calibration while the marking device becomes an obstacle atthe time of work.

SUMMARY

An advantage of some aspects of the invention is to solve at least apart of the problems described above, and the invention can beimplemented as the following configurations.

A control device according to an application example includes areception unit that receives information on a captured image from animaging unit capable of capturing an image, and a control unit thatcontrols driving of a robot including a robot arm on which a markerforming unit that forms a marker on a marker forming object and aworking unit that performs work on a work target are allowed to beprovided by being replaced each other, and the control unit is able toexecute correlation between a robot coordinate system that is acoordinate system relating to the robot and an image coordinate systemthat is a coordinate system relating to the captured image, and thecontrol unit performs the correlation based on a plurality ofcoordinates of a predetermined portion of the robot arm in the robotcoordinate system and a plurality of coordinates of the plurality ofmarkers in the image coordinate system when the plurality of markers areformed on the marker forming object by the marker forming unit, in acase where the working unit is not provided on the robot arm.

According to such a control device, it is possible to performcorrelation (calibration) using a robot including only a marker formingunit without including a working unit. For that reason, since vibrationand the like of the robot arm may be reduced as compared with those inthe related art, it is possible to achieve highly accurate calibration.

A control device according to an application example includes areception unit that receives information on a captured image from animaging unit capable of capturing an image and a control unit that has afunction of performing work on a work target and controls driving of arobot that includes a robot arm on which a marker forming unit thatforms a marker together with a marker forming object is provided, andthe control unit is able to perform correlation between a robotcoordinate system that is a coordinate system relating to the robot andan image coordinate system that is a coordinate system relating to thecaptured image, and the control unit performs the correlation based on aplurality of coordinates of a predetermined portion of the robot arm inthe robot coordinate system and a plurality of coordinates of theplurality of markers in the image coordinate system when the pluralityof markers are formed on the marker forming object by the marker formingunit and the marker forming object.

According to such a control device, since vibration and the like of therobot arm may be reduced as compared with those in the related art, itis possible to achieve highly accurate correlation (calibration). Sincea robot that includes a marker forming unit having a function as aworking unit is used, it is possible to perform calibration and work ofthe robot on a work target using the calibration result more exactly,quickly and accurately.

In the control device according to the application example, it ispreferable that the control unit forms the marker by bringing the markerforming unit into contact with the marker forming object.

With this configuration, it is possible to easily form the marker on themarker forming object.

In the control device according to the application example, it ispreferable that the control unit controls driving of the robot so as toform the marker by bringing the marker forming unit configured by astamp which impresses the marker into contact with the marker formingobject.

With this configuration, it is possible to particularly easily form themarker on the marker forming object. It is possible to configure themarker forming unit to be simple and lightweight and it is possible tofurther reduce the vibration of the robot arm.

In the control device according to the application example, it ispreferable that the control unit controls driving of the robot so as toform the marker by bringing the marker forming unit into contact withthe marker forming object configured by pressure sensitive paper.

With this configuration, it is possible to easily form the marker onpressure sensitive paper by bringing the marker forming unit intocontact with the pressure sensitive paper.

In the control device according to the application example, it ispreferable that the reception unit is able to receive an output from aforce detection device provided in the robot arm and the control unitdetects contact between the marker forming object and the marker formingunit based on the output from the force detection device.

With this configuration, since a contact state at a plurality of placesmay be made uniform or nearly uniform, it is possible to further improvethe accuracy of the calibration.

In the control device according to the application example, it ispreferable that the control unit obtains a relative relationship betweenthe robot coordinate system and the image coordinate system based on aplurality of coordinates in the robot coordinate system and a pluralityof coordinates of the plurality of markers in the image coordinatesystem when the plurality of markers are formed and then, performs thecorrelation based on the relative relationship.

With this configuration, it is possible to further improve accuracy ofthe calibration. Since a predetermined portion of the robot arm may beautomatically moved without a jog operation, for example, it is possibleto simplify work by an operator in the calibration.

In the control device according to the application example, it ispreferable that the control unit performs the correlation usingcoordinates in the robot coordinate system of which the number is largerthan the number of the coordinates in the robot coordinate system usedwhen obtaining the relative relationship and coordinates in the imagecoordinate system of which the number is larger than the number of thecoordinates in the image coordinate system used when obtaining therelative relationship.

With this configuration, it is possible to further improve accuracy ofthe calibration as compared with the case where processing for obtainingthe relative relationship is not performed.

In the control device according to the application example, it ispreferable that the control unit controls driving of the robot so as toform the marker by the marker forming unit provided at a tip end portionof the robot arm.

With this configuration, it is easy to move the marker forming unit to atarget portion, and therefore it is possible to perform the calibrationmore quickly and appropriately.

A robot system according to an application example includes the controldevice of the application example and a robot controlled by the controldevice.

According to such a robot system, it is possible to perform calibrationand work of the robot on a work target to be performed using thecalibration result more exactly, quickly, and accurately.

A control device according to an application example includes aprocessor that controls driving of a robot including a robot arm onwhich a marker forming unit that forms a marker on a marker formingobject and a working unit that performs work on a work target areallowed to be provided by being replaced each other, and the processoris able to execute correlation between a robot coordinate system that isa coordinate system relating to the robot and an image coordinate systemthat is a coordinate system relating to the captured image from theimaging unit capable of capturing an image, and the processor performsthe correlation based on a plurality of coordinates of a predeterminedportion of the robot arm in the robot coordinate system and a pluralityof coordinates of the plurality of markers in the image coordinatesystem when the plurality of markers are formed on the marker formingobject by the marker forming unit, in a case where the working unit isnot provided on the robot arm.

According to such a control device, since calibration using a robot in astate where the working unit is not provided is possible, vibration andthe like of the robot arm may be reduced as compared with those in therelated art and accordingly, it is possible to achieve highly accuratecalibration.

A control device according to an application example includes aprocessor that has a function of performing work on a work target andcontrols driving of a robot that includes a robot arm on which a markerforming unit that forms a marker together with a marker forming objectis provided, and the processor is able to perform correlation between arobot coordinate system that is a coordinate system relating to therobot and an image coordinate system that is a coordinate systemrelating to the captured image from the imaging unit capable ofcapturing an image, and the processor performs the correlation based ona plurality of coordinates of a predetermined portion of the robot armin the robot coordinate system and a plurality of coordinates of theplurality of markers in the image coordinate system when the pluralityof markers are formed on the marker forming object by the marker formingunit and the marker forming object.

According to such a control device, since vibration and the like of therobot arm may be reduced as compared with those in the related art, itis possible to achieve highly accurate calibration. Since a robot thatincludes a marker forming unit having a function as a working unit isused, it is possible to perform calibration and work of the robot on awork target using the calibration result more exactly, quickly, andaccurately.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a diagram illustrating a robot system according to a firstembodiment.

FIG. 2 is a schematic diagram of a robot illustrated in FIG. 1.

FIG. 3 is a diagram illustrating the robot system that includes a robotincluding a working unit.

FIG. 4 is a block diagram illustrating the robot system.

FIG. 5 is a block diagram illustrating a hardware configuration of acontrol device.

FIG. 6 is a flowchart illustrating a control method of the robot by thecontrol device.

FIG. 7 is a flowchart illustrating a flow of calibration.

FIG. 8 is a diagram for explaining step S11.

FIG. 9 is another diagram for explaining step S11.

FIG. 10 is a diagram for explaining step S13.

FIG. 11 is a diagram illustrating a captured image in step S14.

FIG. 12 is a flowchart illustrating the flow of calibration in a secondembodiment.

FIG. 13 is a diagram for explaining step S21.

FIG. 14 is another diagram for explaining step S21.

FIG. 15 is a diagram for explaining step S22.

FIG. 16 is a diagram illustrating a captured image in step S13.

FIG. 17 is a diagram illustrating another example of the captured imagein step S13.

FIG. 18 is a diagram illustrating a robot system according to a thirdembodiment.

FIG. 19 is a diagram for explaining step S11.

FIG. 20 is a diagram for explaining step S11.

FIG. 21 is a diagram illustrating a captured image in step S14.

FIG. 22 is a block diagram illustrating another example of the robotsystem.

FIG. 23 is a block diagram illustrating another example of the robotsystem.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, a control device and a robot system according to theinvention will be described in detail based on preferred embodimentsillustrated in the accompanying drawings.

First Embodiment

FIG. 1 is a diagram illustrating a robot system according to a firstembodiment. FIG. 2 is a schematic diagram of the robot illustrated inFIG. 1. FIG. 3 is a diagram illustrating a robot system that includes arobot including a working unit. FIG. 4 is a block diagram illustrating arobot system. FIG. 5 is a block diagram illustrating a hardwareconfiguration of the control device. In FIG. 1, three axes (X-axis,Y-axis and Z-axis) orthogonal to each other are illustrated. In thefollowing description, a direction parallel to the X-axis is alsoreferred to as an “X-axis direction”, a direction parallel to the Y-axisis also referred to as a “Y-axis direction”, and a direction parallel tothe Z-axis is also referred to as a “Z-axis direction”. In the followingdescription, the tip end side of each arrow illustrated in the drawingis referred to as “+ (plus)” and the base end side is referred to as “−(minus)”. Also, the Z-axis direction coincides with the “verticaldirection” and the direction parallel to the X-Y plane coincides withthe “horizontal direction”. Also, it is assumed that the + (plus) sideof the Z-axis is “upper” and the − (minus) side of the Z-axis is“lower”. In FIG. 2, illustration of a force detection unit 120 isomitted. Also, in FIG. 8, FIG. 18, and FIG. 19 to be described later,illustration of the force detection unit 1200 is omitted.

In the present specification, the term “horizontal” includes a casewhere it is inclined within a range of ±10° or less with respect to thehorizontal. Similarly, the term “vertical” includes a case where it isinclined within a range of ±10° or less with respect to the vertical.The term “parallel” includes not only a case where two lines (includingaxes) or a plane are perfectly parallel to each other but also a casewhere it is inclined within ±10°. The term “orthogonal” includes notonly a case where two lines (including axes) or a plane intersect at anangle of 90° with each other but also a case of being inclined within±10° with respect to 90°. Further, in the present specification, theterm “connection” includes a case of being connected directly and a caseof being connected indirectly to any member.

Robot System

A robot system 100 illustrated in FIG. 1 can be used, for example, forholding, transporting, assembling a work target 91 such as an electroniccomponent and the like. The robot system 100 includes a robot 1 and acontrol device 5 that controls driving of the robot 1. In the robotsystem 100, an imaging unit 3 having an imaging function, a displaydevice 401 including a monitor, and an input device 402 (operationdevice) configured by, for example, a mouse, a keyboard, and the likeare connected so as to be able to communicate with each other (see FIG.4).

Robot

As illustrated in FIGS. 1 and 2, the robot 1 is a so-called six-axisvertical articulated robot, and includes a base 110 and a robot arm 10connected to the base 110.

The base 110 is a portion for attaching the robot 1 to any installationplace. In the first embodiment, the base 110 is installed at aninstallation place 70 such as a floor. The installation place of thebase 110 is not limited to the installation place 70 such as a floor,but may be, for example, a wall, a ceiling, on a movable carriage, orthe like.

As illustrated in FIGS. 1 and 2, the robot arm 10 includes an arm 11(first arm), an arm 12 (second arm), an arm 13 (third arm), an arm 14(fourth arm), an arm 15 (fifth arm), and an arm 16 (sixth arm, tip endarm). These arms 11 to 16 are connected in this order from the proximalend side to the tip end side. Each of the arms 11 to 16 is rotatablewith respect to an adjacent arm or the base 110. Here, as illustrated inFIG. 1, the arm 16 has a disk shape and is rotatable around a rotationaxis O6 with respect to the arm 15. As illustrated in FIG. 2, in thepresent embodiment, the center of the tip end surface of the arm 16 isreferred to as a predetermined point P6 (predetermined portion).

A stamp 19 (marker forming unit) is detachably attached to the tip endof such a robot arm 10 via a force detection unit 120 to be describedlater. The stamp 19 is a marker forming unit for forming a marker 190 ona work surface 710 of a work stand 71 (see FIG. 9) and is a componentfor forming the marker 190 on the work surface 710. The marker 190 is amark configured by a letter, a mark such as a figure, and the like. Inthe first embodiment, the stamp 19 is engraved with a circular mark onthe tip end surface, a coloring agent such as ink is applied to the tipend surface, and the stamp 19 is pressed against the work surface 710,thereby making it possible to leave a circular imprint on the worksurface 710. The work stand 71 (marker forming object) is a stand onwhich the work target 91 can be placed, and the work stand 71(particularly, work surface 710) is made of a material capable offorming the marker 190, for example, paper, wood, resin, a metal, or thelike.

In the first embodiment, the center of the tip end of the stamp 19 isreferred to as a tool center point P9. In the first embodiment, the toolcenter point P9 and the predetermined point P6 described above are onthe rotation axis O6.

Instead of the stamp 19, a hand 17 (working unit) for gripping the worktarget 91 can be attached to the robot arm 10 (specifically, robot arm10 including force detection unit 120) (see FIG. 1 and FIG. 3). That is,the robot arm 10 is configured so that the stamp 19 and the hand 17 canbe replaced with each other. For example, the robot arm 10 may have aconfiguration in which a female screw or a male screw used for attachingthe stamp 19 or the hand 17 by screwing, bolting, or the like isincluded, or a configuration in which an engaging portion such as a hookand an L-shaped groove (not illustrated) is included. With this, it ispossible to easily attach the stamp 19 or the hand 17 to an appropriateposition and to easily replace the stamp 19 and the hand 17 each other.The hand 17 is configured to include, for example, a metal material. Inthe first embodiment, the center of the tip end of the hand 17 (centerbetween two fingers) is referred to as a tool center point P7. In thefirst embodiment, the tool center point P7 and the predetermined pointP6 described above are on the rotation axis O6.

As illustrated in FIG. 1, between the arm 16 and the stamp 19, a forcedetection unit 120 (force detection device) is detachably provided tothe arm 16 and the stamp 19. The force detection unit 120 detects aforce (including moment) applied to the stamp 19. The force detectionunit 120 is configured by, for example, a six-axis force sensor, athree-axis force sensor, or the like. The force detection unit 120outputs detected force detection information to the control device 5.

As illustrated in FIG. 4, the robot 1 includes a driving unit 130including a motor, a reduction gear, and the like for rotating one armwith respect to the other arm (or base 110). As the motor, for example,a servo motor such as an AC servo motor, a DC servo motor or the likecan be used. As the reduction gear, for example, a planetary gear typereduction gear, a wave gear device or the like can be used. The robot 1includes a position sensor 140 (angle sensor) for detecting a rotationangle of a rotation shaft of the motor or the reduction gear. As theposition sensor 140, for example, a rotary encoder or the like can beused. The driving unit 130 and the position sensor 140 are provided, forexample, in the respective arms 11 to 16, and in the first embodiment,the robot 1 includes six driving units 130 and six position sensors 140.Each of the driving units 130 is electrically connected to the controldevice 5 via, for example, a motor driver (not illustrated) built in therobot 1. Each position sensor 140 is also electrically connected to thecontrol device 5.

In such a robot 1, a base coordinate system (robot coordinate system)based on the base 110 of the robot 1 is set. The base coordinate systemis a three-dimensional orthogonal coordinate system defined by theX-axis and Y-axis parallel to the horizontal direction and the Z-axisorthogonal to the horizontal direction and having the vertically upwarddirection as the positive direction. In the first embodiment, the basecoordinate system has the center point of the upper end surface of thebase 110 as the origin. It is assumed that a translational componentwith respect to the X-axis is a “component X”, the translationalcomponent with respect to the Y-axis is a “component Y”, and thetranslational component with respect to the Z-axis is a “component Z”.The unit of length (size) of the component X, component Y and componentZ is “mm”.

Further, in the robot 1, a tip end coordinate system having thepredetermined point P6 of the arm 16 as the origin is set. The tip endcoordinate system is a two-dimensional orthogonal coordinate systemdefined by the Xa-axis and Ya-axis orthogonal to each other. Each of theXa-axis and the-Ya axis is orthogonal to the rotation axis O6. Further,it is assumed that the translational component with respect to theXa-axis is a “component Xa”, and the translational component withrespect to the Ya axis is a “component Ya”. The unit of the length(size) of the component Xa and the component Ya is “mm”. In the firstembodiment, calibration of the base coordinate system and the tip endcoordinate system has been completed, and the coordinates in the tip endcoordinate system can be obtained from calculation from the coordinatesin the base coordinate system. In the first embodiment, the basecoordinate system is taken as a “robot coordinate system”, but the tipend coordinate system may be assumed as the “robot coordinate system”.

The configuration of the robot 1 has been briefly as described above. Inthe first embodiment, as described above, although a case where the“working unit” is the hand 17 has been described as an example, the“working unit” may be anything as long as it performs any work on thework target 91 other than work of forming the marker 190 and may be, forexample, a device (not illustrated) provided with an adsorptionmechanism, a device for performing screw fastening, or the like. Thestamp 19 described above performs work of forming the marker 190 on thework target 91, unlike the “working unit”.

Imaging Unit

As illustrated in FIG. 1 and FIG. 2, the imaging unit 3 is positionedvertically above the installation place 70 and is installed so as tomake it possible to image the work surface 710 of the work stand 71.

Although not illustrated, the imaging unit 3 includes, for example, animaging device configured by a charge coupled device (CCD) image sensorhaving a plurality of pixels, and an optical system including a lens.The imaging unit 3 forms an image of light from an imaging target or thelike onto a light receiving surface of an imaging element with a lens,converts light into an electric signal, and outputs the electric signalto the control device 5. The imaging unit 3 is not limited to theconfiguration described above as long as it has an imaging function, andother configurations may be adopted.

In such an imaging unit 3, an image coordinate system, that is, acoordinate system of the captured image 30 output from the imaging unit3 is set (see FIG. 11). This image coordinate system is atwo-dimensional orthogonal coordinate system determined by the U-axisand V-axis respectively parallel to the in-plane direction of thecaptured image 30. In the first embodiment, it is assumed that thetranslational component with respect to the U-axis is a “component U”and the translational component with respect to the V axis is a“component V”. The unit of the length (magnitude) of the component U andthe component V is a “pixel”. The image coordinate system is atwo-dimensional orthogonal coordinates obtained by nonlinearlyconverting three-dimensional orthogonal coordinates imaged in the fieldof view of a camera of the imaging unit 3 by taking into considerationthe optical characteristics (focal length, distortion, and the like) ofthe lens and the number of pixels and size of the image pickup element.

Control Device

As illustrated in FIG. 4, the control device 5 has a function ofcontrolling driving of the robot 1 and is connected to the robot 1 andthe imaging unit 3 so as to be able to communicate with each other. Thecontrol device 5, the robot 1, and the imaging unit 3 may be connectedwith each other through wired connection or wireless connection. Thecontrol device 5 is connected to a display device 401 including amonitor (not illustrated) and an input device 402 configured by, forexample, a keyboard and the like.

As illustrated in FIG. 4, the control device 5 includes a control unit51 including a processor, a storing unit 52 including a memory and thelike, and an external input and output unit 53 including an externalinterface (I/F). The respective constituent elements of the controldevice 5 are connected so as to communicate with each other via variousbuses.

The control unit 51 (processor) includes a processor such as a centralprocessing unit (CPU) and executes various programs and the like storedin the storing unit 52 (memory). With this, it is possible to realizecontrol of driving of the robot 1 and processing such as variousoperations and determination.

In the storing unit 52, various programs executable by the control unit51 are stored. In the storing unit 52, various data received by theexternal input and output unit 53 can be stored. The storing unit 52 isconfigured to include a volatile memory such as a random access memory(RAM), a nonvolatile memory such as a read only memory (ROM), and thelike. The storing unit 52 is not limited to a non-detachable type andmay be configured to include a detachable external storage device (notillustrated).

Various programs include a robot drive command relating to driving ofthe robot 1, an image coordinate conversion command relating tocorrelation between the image coordinate system and the tip endcoordinate system of the robot 1 or the robot coordinate system (basecoordinate system), a robot coordinate conversion command relating tocorrelation between the tip end coordinate system and the basecoordinate system, and the like.

The image coordinate conversion command is a command for obtaining acoordinate conversion expression for converting image coordinates (U, V:position), which are coordinates in the image coordinate system, intothe robot coordinates (X, Y: position) which are the coordinates in therobot coordinate system or the tip coordinates (Xa, Ya: position) whichare the coordinates in the tip end coordinate system. This imagecoordinate conversion command is executed so as to make it possible toperform correlation (calibration) between the image coordinate systemand the robot coordinate system and the tip end coordinate system.

As various types of data, for example, data output from a plurality ofposition sensors 140 included in the robot 1 and data relating to thecaptured image 30 output from the imaging unit 3 are included. Asvarious types of data, data such as the number of pixels of the imagingunit 3 and data relating to the speed and acceleration (morespecifically, moving speed and movement acceleration of the stamp 19,for example) of the robot 1 at the time of execution of calibration, andthe like are included.

The control unit 51 executes the program stored in the storing unit 52so as to make it possible to convert the rotational component(orientation in the image coordinate system) around the normal line ofthe U-V plane into the rotation component (orientation in the robotcoordinate system) around the normal line of the X-Y plane or therotation component around the normal line of the Xa-Ya plane(orientation in the tip end coordinate system). For example, the valueof the rotation angle of the work target 91 (or marker 190) with respectto model (template) of a target contour shape registered in advance isacquired by comparing the work target 91 or the like captured with thecaptured image 30 with the model. With this, it is possible to obtainthe orientation of the work target 91 in the image coordinate system.The orientation of the work target 91 in the robot coordinate system andthe orientation of the work target 91 in the tip end coordinate systemcan be obtained based on correlation between the image coordinatesystem, the tip end coordinate system and the robot coordinate system.

The external input and output unit 53 includes an external interface(I/F), and is used for each connection of the robot 1, the imaging unit3, the display device 401, and the input device 402. The external inputand output unit 53 functions as a “reception unit” that receivesinformation (data) on the captured image 30 from the imaging unit 3.

Such a control device 5 is configured to include, for example, acontroller 61 communicably connected to the robot 1 and a computer 62communicably connected to the controller 61 as illustrated in FIG. 5.The control device 5 may be configured by the controller 61. Control ofdriving of the robot 1 may be executed by reading a command (program ordata) present in the memory by a processor present in the controller 61,or may be executed via the controller 61 by reading a command in thememory by a processor present in the computer 62.

The control device 5 may have a configuration to which anotherconfiguration is added, in addition to the configuration describedabove. Various programs, data, and the like stored in the storing unit52 (memory) may be stored in the storing unit 52 in advance, or may bestored in a recording medium (not illustrated) such as a CD-ROM, may beprovided from this recording medium, or may be provided via a network orthe like.

Display Device and Input Device

The display device 401 illustrated in FIG. 4 includes a monitor and hasa function of displaying various screens and the like. Accordingly, theoperator can confirm the captured image 30 outputted from the imagingunit 3 and driving of the robot 1 via the display device 401.

The input device 402 is configured by, for example, a keyboard or thelike. Accordingly, the operator operates the input device 402 so as tomake it possible to issue instructions, such as instructions to executevarious processing, to the control device 5. Although not illustrated,the input device 402 may be configured by, for example, a teachingpendant.

Instead of the display device 401 and the input device 402, a displayinput device (not illustrated) having both functions of the displaydevice 401 and the input device 402 may be used. As the display inputdevice, for example, a touch panel display or the like can be used. Therobot system 100 may have one display device 401 and one input device402, or may have a plurality of display devices 401 and a plurality ofplural input devices 402.

The basic configuration of the robot system 100 has been brieflydescribed as above. As described above, the robot system 100 includesthe control device 5 and the robot 1 controlled by the control device 5.Then, the control device 5 executes control to be described later.

According to such a robot system 100, since control by the controldevice 5 to be described later can be executed, it is possible toperform calibration and work on the work target 91 of the robot 1 to beperformed using the calibration result more exactly, quickly andaccurately.

Control Method

FIG. 6 is a flowchart illustrating a control method of the robot by thecontrol device.

As illustrated in FIG. 6, the control method of the robot 1 by thecontrol device 5 includes a step of calibration (step S10) and a step ofwork by the robot 1 based on the result of calibration (step S20).

Although the specific work content of the work (step S20) by the robot 1is not particularly limited as long as it is a work content to beperformed on the work target 91 by the hand 17 (working unit), forexample, work of gripping the work target 91 placed on the work stand 72by the hand 17 and transporting and placing the work target 91 on thework stand 71 by the hand 17 (see FIG. 3) is included. Since thespecific work content of the work (step S20) by the robot 1 is notparticularly limited, description thereof will be omitted below anddescription will be made on the calibration (step S10) below.

Calibration

FIG. 7 is a flowchart illustrating a flow of the calibration. FIG. 8 andFIG. 9 are diagrams for explaining step S11, respectively. FIG. 10 is adiagram for explaining step S13. FIG. 11 is a diagram illustrating acaptured image in step S14.

In the calibration (step S10), calibration (correlation) between theimage coordinate system of the imaging unit 3 and the robot coordinatesystem of the robot 1 is performed. Specifically, in order to cause therobot 1 to perform various operations based on data of the capturedimage 30 output from the imaging unit 3, a coordinate conversionexpression (coordinate transformation matrix) for converting coordinates(image coordinates) in the image coordinate system into coordinates(robot coordinates) in the robot coordinate system is obtained. That is,obtaining the coordinate conversion expression to make it possible toobtain the robot coordinates from the image coordinates by calculationis equivalent to “correlation” between the image coordinate system andthe robot coordinate system.

In the first embodiment, calibration is performed using the stamp 19illustrated in FIG. 1. The stamp 19 is a component for forming themarker 190 and does not have the function of holding the work target 91.In the first embodiment, the tool center point P9 and the predeterminedpoint P6 are on the rotation axis O6, and setting (tool setting on theX-axis, the Y-axis, and the Z-axis) of the position of the tool centerpoint P9 with respect to the predetermined point P6, that is, thecorrelation between the predetermined point P6 and the tool center pointP9 is completed, and the coordinates of the tool center point P9 withrespect to the predetermined point P6 can be obtained from thecalculation.

Hereinafter, the calibration will be described with reference to theflowchart illustrated in FIG. 7. This calibration is performed by thecontrol unit 51 executing the program stored in the storing unit 52 inaccordance with an instruction made by the operator using the inputdevice 402.

First, the control unit 51 drives the robot arm 10 to position the stamp19 within the field of view of the imaging unit 3, that is, within animaging area S3 as illustrated in FIG. 8, and forms the marker 190 onthe work surface 710 of the work stand 71 (Step S11). With this, acircular marker 190 is formed on the work surface 710 of the work stand71 (see FIG. 9). The robot arm 10 for forming the marker 190 is drivenby, for example, a jog operation. The jog operation is an operation ofthe robot 1 based on an instruction of guidance by the operator usingthe input device 402 such as a teaching pendant.

Next, the control unit 51 stores the robot coordinates of thepredetermined point P6 when the marker 190 is formed by the stamp 19 inthe storing unit 52 (step S12).

Next, the control unit 51 determines whether or not the number of timesin which step S12 described above has been performed reaches apredetermined number of times (step S13), and repeats steps S11 and S12until the predetermined number of times is reached. In the firstembodiment, the predetermined number of times is nine. Accordingly, inthe first embodiment, the control unit 51 repeats steps S11 and S12until it is determined that nine robot coordinates have been acquired.

Here, the control unit 51 moves the stamp 19 so that the marker 190 ispositioned within the imaging area S3 of the work surface 710 in stepS11 at each time. The control unit 51 moves the stamp 19 so that themarkers 190 are formed at different positions at each time. Inparticular, as illustrated in FIG. 10, the control unit 51 preferablymoves the stamp 19 such that the plurality of markers 190 are arrangedin a lattice pattern. Accordingly, for example, the control unit 51forms the marker 190 at the upper left position (first position P10) inFIG. 10 in the first step S11 and forms the marker 190 at the position(second position P 20) at the center on the left side in FIG. 10 in thesecond step S11. The control unit 51 stores the robot coordinates in thestoring unit 52 in each of the first and second rounds. By doing asdescribed above, the control unit 51 repeats steps S11 and S12 ninetimes and stores the robot coordinates of nine predetermined points P6in the storing unit 52.

Next, when it reaches a predetermined number of times (nine times in thefirst embodiment), the control unit 51 causes the imaging unit 3 tocollectively capture the nine markers 190 and stores the imagecoordinates of the respective markers 190 in the storing unit 52 (StepS14). FIG. 11 illustrates the captured image 30 at this time.

Next, based on the robot coordinates of nine predetermined points P6 andthe image coordinates of the nine markers 190, the control unit 51obtains the coordinate conversion expression for converting the imagecoordinates into the robot coordinates (step S15). With this, thecalibration between the image coordinate system and the robot coordinatesystem, that is, correlation is completed.

Processing of the calibration (step S10) has been briefly described asabove. By using the coordinate conversion expression obtained byprocessing of the calibration (step S10), the position and orientationof the imaging target imaged by the imaging unit 3 can be converted intothe position and orientation in the robot coordinate system.Furthermore, as described above, since the robot coordinate system (basecoordinate system) and the tip end coordinate system have already beencorrelated with each other, it is possible to convert the position andorientation of the imaging target imaged by the imaging unit 3 into theposition and orientation in the tip coordinate system. For that reason,the control unit 51 can position the stamp 19 at a target place based onthe captured image 30. Furthermore, when the stamp 19 is replaced withthe hand 17, the control unit 51 can position the hand 17 at the targetplace based on the captured image 30. Therefore, by using the coordinateconversion expression between the robot coordinate system and the imagecoordinate system, which is the result in the calibration (step S10), itis possible to cause the robot 1 to work appropriately using the hand 17in the work by the robot 1 (step S20 in FIG. 6).

In the first embodiment, although the tool center point P7 of the hand17 is on the rotation axis O6, in a case where the tool center point P7is not on the rotation axis O6, it is preferable to set (tool setting)the position of the tool center point P7 with respect to thepredetermined point P6 after replacing the stamp 19 with the hand 17.With this, it is possible to perform the work by the robot 1 morequickly and accurately based on the result of the calibration.

The control method has been described as above. As described above, thecontrol device 5 includes the external input and output unit 53 (I/F)having a function as a “reception unit” that receives information on thecaptured image 30 from the imaging unit 3 that can capture images, andthe control unit 51 (processor) that controls driving of the robot 1(which can execute instructions relating to driving) including the robotarm 10 on which the stamp 19 (marker forming unit) that forms the marker190 on the work stand 71 (marker forming object) and the hand 17(working unit) that performs work on the work target 91 are allowed tobe provided by being replaced each other (see FIGS. 1 to 4 and thelike). Further, the control unit 51 can execute correlation(calibration) between the robot coordinate system that is the coordinatesystem relating to the robot 1 and the image coordinate system that isthe coordinate system relating to the captured image 30. When the hand17 is not provided on the robot arm 10, the control unit 51 performs thecorrelation based on the plurality of robot coordinates (coordinates) ofthe predetermined point P6 (predetermined portion) of the robot arm 10in the robot coordinate system when the plurality of markers 190 areformed on the work stand 71 by the stamp 19 and the plurality of imagecoordinates (coordinates) of the plurality of markers 190 in the imagecoordinate system.

According to such a control device 5, since the robot 1 can performcalibration (correlation) using the robot 1 which is in a state wherethe hand 17 is not attached and in which the stamp 19 is attached,weight of the tip end of the robot arm 10 can be greatly reduced ascompared with the case of performing the calibration using the robot 1which is in a state where both of the hand 17 and the stamp 19 aremounted. For that reason, since the vibration and the like of the robotarm 10 can be reduced as compared with that of the related art, it ispossible to achieve highly accurate calibration. Since it is possible toperform the calibration using the marker 190 formed at any place, it isnot necessary to perform touch-up processing unlike the related art. Forthat reason, it is possible to perform the calibration more quickly.Furthermore, since the image coordinates of each marker 190 are obtainedbased on the captured image 30 obtained by collectively imaging theplurality of markers 190, it is possible to perform the calibration ascompared with the case where the captured image 30 is acquired each timethe marker 190 is formed. Since it is possible to cause the robot 1 toperform various work to be conducted based on the calibration result(coordinate conversion expression) using the work target 91, it ispossible to cause the robot 1 to perform work on the work target 91accurately. The hand 17 becomes an obstacle at the time of calibrationand the marking device does not become an obstacle at the time of workand accordingly, the calibration and work can be performed quickly andaccurately.

In the first embodiment, although the predetermined point P6 is set asthe predetermined portion, the predetermined portion may be any place ofthe robot arm 10. For example, the predetermined portion may be thecenter of the tip end of the arm 15.

As described above, the control unit 51 forms the marker 190 by bringingthe stamp 19 (marker forming unit) into contact with the work stand 71(marker forming object).

With this, the marker 190 can be easily formed on the work stand 71. Forexample, in a case where the marker forming unit is a device or the likethat outputs a laser, a marker can be formed without bringing the markerforming unit into contact with the marker forming object.

In particular, in the first embodiment, as described above, the markerforming unit is configured by the stamp 19. Then, the control unit 51controls driving of the robot 1 so as to form the marker 190 by bringingthe marker forming unit configured by the stamp 19 to which the marker190 is attached into contact with the work stand 71 (marker formingobject).

With this, the marker 190 can be particularly easily formed on the workstand 71. Since the stamp 19 has a simple and lightweight construction,vibration of the robot arm 10 can be further reduced. For that reason,it is possible to further improve accuracy of the calibration by usingthe stamp 19.

In the first embodiment, the work stand 71 configured to include, forexample, paper, wood, resin, metal, or the like, is used as the markerforming object, but the marker forming object may be anything as long asthe marker forming unit can form a marker. In the first embodiment, thestamp 19 is used as the marker forming unit, but the marker forming unitmay be anything as long as it is capable of forming a marker withrespect to the marker forming object. For example, the marker formingobject may be a touch panel display including a sensor that detects atouch, and the marker forming unit may be an object (for example, apen-like object) of which touch can be detected on the touch paneldisplay. For example, the marker forming unit may be a writinginstrument such as a pen.

As described above, the force detection unit 120 (force detectiondevice) is provided in the robot arm 10 is with. The external input andoutput unit 53 having the function as the “reception unit” can receivethe output from the force detection unit 120 (force detection device)provided in the robot arm 10, and the control unit 51 detects contactbetween the work stand 71 (marker forming object) and the stamp 19(marker forming unit) based on the output from the force detection unit120.

With this, since a contact state at a plurality of places can be madeuniform or nearly uniform, it is possible to further improve theaccuracy of the calibration.

As described above, the stamp 19 is provided on the tip end portion ofthe robot arm 10, that is, the arm 16 positioned at the extreme end.Then, the control unit 51 controls driving of the robot 1 so as to formthe marker 190 with the stamp 19 (marker forming unit) provided at thetip end portion of the robot arm 10.

With this, it is easy to move the stamp 19 to the intended place,thereby making it possible to perform the calibration more quickly andappropriately.

The stamp 19 (marker forming unit) may be provided, for example, on thearm 15 or the arm 14. In the first embodiment, the robot 1 is aso-called six-axis vertical articulated robot, but the “robot”controlled by the control device 5 may be a SCALA robot (notillustrated). In the case of using the SCALA robot, though notillustrated, the marker forming unit may be provided on a shaft(operating shaft) provided on the arm included in the SCALA robot, ormay be provided on an arm provided with a shaft. As a specific example,although not illustrated, in a case where the SCARA robot includes abase, a first arm (arm) connected to the base, a second arm (arm)connected to the first arm, and a shaft (operating shaft) provided onthe second arm, the marker forming unit may be provided on the shaft ormay be provided on the second arm.

As described above, the imaging unit 3 is installed so as to be able toimage the work surface 710 of the work stand 71. The external input andoutput unit 53 is capable of communicating with the imaging unit 3provided so as to be able to image the work stand 71 on which the worktarget 91 is disposed.

With this, the marker 190 formed on the work stand can be imaged, andthe calibration can be accurately performed using the captured image 30.Furthermore, even when work is performed on the work target 91 by therobot 1, the robot 1 can appropriately perform the work using thecaptured image 30.

In the first embodiment, as illustrated in FIG. 6, although work by therobot 1 (step S20) is performed after the calibration (step S10), if theresult of the calibration is used in step S20, Step S20 may be performedalone. The calibration (step S10) may be performed alone.

Second Embodiment

Next, a second embodiment will be described.

FIG. 12 is a flowchart illustrating the flow of calibration in a secondembodiment. Each of FIGS. 13 and 14 is a diagram for explaining stepS21. FIG. 15 is a diagram for explaining step S22. FIG. 16 is a diagramillustrating a captured image in step S13. FIG. 17 is a diagramillustrating another example of the captured image in step S13.

The second embodiment is the same as the first embodiment describedabove except that a relative relationship between the robot coordinatesystem and the image coordinate system is mainly obtained and steps S11to S16 are automatically performed. In the following description,description will be mainly made on differences from the first embodimentdescribed above, and description of similar matters will be omitted.

Hereinafter, the calibration in the second embodiment will be describedwith reference to the flowchart illustrated in FIG. 12.

First, prior to performing step S11 described above, the control unit 51obtains the relative relationship between the robot coordinate systemand the image coordinate system (step S21).

Specifically, first, the control unit 51 forms three markers 190 atdifferent positions on the work surface 710 within the field of view ofthe imaging unit 3 as illustrated in FIG. 13, and then, acquires thecaptured image 30 as illustrated in FIG. 14 and acquires the robotcoordinates and image coordinates in the three markers 190. Next, thecontrol unit 51 obtains coefficients a, b, c, and d in the followingequation (1,) based on the acquired three robot coordinates and theacquired three image coordinates acquired. With this, a coordinateconversion expression between the robot coordinates and the imagecoordinates can be obtained, and it is possible to convert displacement(movement amount) in the image coordinate system into a displacementamount in the robot coordinate system (base coordinate system) orfurther into displacement (movement amount) in the tip end coordinatesystem.

$\begin{matrix}{\begin{pmatrix}{\Delta\; U} \\{\Delta\; V}\end{pmatrix} = {\begin{pmatrix}a & b \\c & d\end{pmatrix}\begin{pmatrix}{\Delta\; X} \\{\Delta\; Y}\end{pmatrix}}} & (1)\end{matrix}$

ΔX and ΔY in the expression (1) indicate displacement (movement amount)between two places in the image coordinate system, and ΔU and ΔVindicate displacement (movement amount) between two places in the robotcoordinate system.

For example, it is assumed that the movement amount of the predeterminedpoint P6 to the point at the time of forming a marker 190 b by moving inthe direction of an arrow R1 from the point at the time of forming amarker 190 a (reference point) is 10 mm in the X-direction and 0 mm inthe Y-direction, and the movement amount of the predetermined point P6to the point at the time of forming a marker 190 c by moving in thedirection of an arrow R2 from the point at the time of forming themarker 190 a is 0 mm in the X-direction and 10 mm in the Y-direction(see FIG. 13). It is assumed that the image coordinates of the marker190 a are (U0, V0), the image coordinates of the marker 190 b is (U1,V1), and the image coordinates of the marker 190 c is (U2, V2) (see FIG.14). In this case, the coefficients a, b, c, and d can be obtained asfollows.ΔU ₁ −U1−U0 ΔU ₁=10·a+0·bΔV ₁ −V1−V0 ΔV ₁=10·c+0·d⇒a=ΔU ₁/10, c=ΔV ₁/10ΔU₂−U2−U0 ΔU₂−0·a+10·bΔV₂−V2−V0 ΔU₂−0·c+10·d⇒b−ΔU₂/10, d−ΔV₂/10

As such, by using the coordinate conversion expression (affineconversion expression) illustrated in the above expression (1) based onthe three robot coordinates and the three image coordinates obtained bymoving the predetermined point P6 while forming the marker 190 at threedifferent places, the relative relationship between the robot coordinatesystem and the image coordinate system can be easily and appropriatelyobtained. The relative relationship means a relationship in which themovement amount in the image coordinate system can be obtained from themovement amount in the robot coordinate system by calculation.

Next, as illustrated in FIG. 15, nine teaching points 301 in thecaptured image 30 are set (step S22). In the second embodiment, nineteaching points 301 arranged in a lattice pattern are set. Specifically,as illustrated in FIG. 15, the control unit 51 divides a search window300 of the captured image 30 into nine parts, and sets the center ofeach divided region as the teaching point 301.

In step S22, after the nine teaching points 301 are set, the controlunit 51 calculates the movement amount (ΔU, ΔV) in the image coordinatesystem from the marker 190 a (reference point) to each teaching point301 (teaching points 301 a to 301 i) is obtained.

For example, the control unit 51 can obtain the movement amount (ΔU, ΔV)in the image coordinate system from the marker 190 a (reference point)to each teaching point 301 as indicated below. The movement amounts (ΔU,ΔV) in the following image coordinate systems correspond to the teachingpoints 301 a, 301 b, 301 c, 301 d, 301 e, 301 f, 301 g, 301 h, and 301 iin order from the top. In the following description, the coordinates ofthe marker 190 a (reference point) are indicated by (Xref, Yref), thelength in the U-axis direction of the search window 300 is indicated by“Width”, the length in the V-axis direction of the search window 300 isindicated by “Height”, the distance in the U-axis direction from theorigin O to the search window 300 is indicated by “Left”, and thedistance in the V-axis direction from the origin O to the search window300 is indicated by “Top”.

$\left( {{\left( {{Left} + \frac{Width}{6}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{6}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{Width}{2}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{6}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{5 \cdot {Width}}{6}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{6}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{5 \cdot {Width}}{6}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{2}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{Width}{2}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{2}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{Width}{6}} \right) - {Xref}},{\left( {{Top} + \frac{Height}{2}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{Width}{6}} \right) - {Xref}},{\left( {{Top} + \frac{5 \cdot {Height}}{6}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{Width}{2}} \right) - {Xref}},{\left( {{Top} + \frac{5 \cdot {Height}}{6}} \right) - {Yref}}} \right)$$\left( {{\left( {{Left} + \frac{5 \cdot {Width}}{6}} \right) - {Xref}},{\left( {{Top} + \frac{5 \cdot {Height}}{6}} \right) - {Yref}}} \right)$

In step S22, based on the coordinate conversion expression obtained instep S21 and the movement amount (ΔU, ΔV) in the image coordinatesystem, the control unit 51 obtains the relative movement amount (ΔX,ΔY) of the predetermined point P6 in the robot coordinate system. Withthis, it is possible to automatically perform steps S11 to S15regardless of the jog operation. In FIG. 16, the captured image 30obtained by imaging the nine markers 190 in step S14 is illustrated. InFIG. 16, three markers 190 formed in step S21 are indicated by brokenlines, and the nine markers 190 formed in step S11 are indicated bysolid lines.

Here, although the control unit 51 forms nine markers 190 (step S11) asindicated by the solid lines in FIG. 16 after the three markers 190 areformed as indicated by broken lines in FIG. 16 (step S21), for example,the control unit 51 may form each marker 190 as illustrated in FIG. 17.Specifically, the control unit 51 may form the nine markers 190 asindicated by the solid lines in FIG. 17 (Step S11) after forming thethree markers 190 as indicated by the broken line in FIG. 17 (step S21).That is, in step S11, the control unit 51 forms nine markers 190 atdifferent positions (not overlapping) from positions of the threemarkers 190 formed in step S21. With this, the markers 190 do notoverlap each other and thus, visibility of the operator can be enhanced.

The control unit 51 may make colors and shapes of the marker 190 to beformed in step S21 different from those of the marker 190 to be formedin step S11. Even with such a method, visibility of the operator can beenhanced.

The calibration in the second embodiment has been described as above. Asdescribed above, after obtaining a relative relationship between therobot coordinate system and the image coordinate system based on aplurality of robot coordinates (coordinates) in the robot coordinatesystem and a plurality of image coordinates (coordinates) of theplurality of markers 190 in the image coordinate system when theplurality of markers 190 are formed, the control unit 51 performs thecorrelation based on the relative relationship.

With this, it is possible to further improve accuracy of thecalibration. As described above, since the movement amount in the imagecoordinate system can be obtained from the movement amount in the robotcoordinate system by obtaining the relative relationship, in step S11,nine teaching points 301 (plurality of places), it is possible toautomatically move the predetermined point P6 to the nine teachingpoints 301 (a plurality of points) regardless of the jog operation instep S11. For that reason, it is possible to easily perform work of theoperator in the calibration.

Furthermore, the control unit 51 performs the correlation using therobot coordinates (coordinates) in the robot coordinate system of whichthe number is larger than the number of robot coordinates (coordinates)in the robot coordinate system used when obtaining the relativerelationship and the image coordinates (coordinates) in the imagecoordinate system of which the number is larger than the number of imagecoordinates (coordinates) in the image coordinate system used whenobtaining the relative relationship. In the second embodiment, threerobot coordinates and three image coordinates are used in step S21, andnine robot coordinates and nine image coordinates are used whenobtaining the coordinate conversion expression in step S15.

With this, since the marker 190 can be formed at substantially evenintervals at nine places (teaching points 301) as compared with the casewhere processing for obtaining the relative relationship is notperformed, it is possible to further improve accuracy of thecalibration.

In the second embodiment, although nine teaching points 301 are present,the number of teaching points 301 is arbitrary and may be at leastthree. However, as the number of the teaching points 301 is increased,accuracy of the calibration is improved. It is preferable that thenumber of teaching points 301 is equal to or more than the number ofrobot coordinates (three in the second embodiment) used when obtainingthe coordinate conversion expression in step S21. In the secondembodiment, although the teaching points 301 are arranged in a latticepattern, the arrangement of teaching points 301 are not limited to thelattice pattern shapes.

In the second embodiment, similarly to the first embodiment describedabove, the case where the tool center point P9 and the predeterminedpoint P6 are on the rotation axis O6 and the robot coordinates (X, Y) ofthe tool center point P9 coincides with the robot coordinates (X, Y) ofthe predetermined point P6 is described by way of an example. That is,the case where setting (tool setting of the X-axis, the Y-axis, and theZ-axis) of the position of the tool center point P9 with respect to thepredetermined point P6, that is, the correlation between thepredetermined point P6 and the tool center point P9 is in a state inwhich the position and orientation (coordinates) of the tool centerpoint P9 with respect to the predetermined point P6 can be obtained fromthe calculation was described by way of an example. In contrast, in acase where the robot coordinates (X, Y) of the tool center point P9 andthe robot coordinates (X, Y) of the predetermined point P6 do notcoincide with each other, it is preferable to set the position of thetool center point P9 with respect to the predetermined point P6 (Xcoordinate and Y coordinate tool setting) after step S22 before stepS11. With this, it is possible to improve accuracy of calibration evenwhen the tool center point P9 and the predetermined point P6 do notcoincide with each other.

Although the method of setting the tool is not particularly limited, forexample, a method of moving the predetermined point P6 to two differentpositions while positioning the marker 190 (tool center point P9) at thecenter of the image of the captured image 30 is included. In thismethod, the position of the tool center point P9 with respect to thepredetermined point P6 is obtained based on the robot coordinates andimage coordinates of the predetermined point P6 before and after themovement, a rotation angle θ of the predetermined point P6 around thetool center point P9, and the coordinates of the marker 190 in the imagecoordinate system. According to such a method, it is possible to set thetool easily and accurately. In a case where the position of the toolcenter point P9 with respect to the predetermined point P6 is obtainedfrom a design value or an actual measurement value, the design value orthe actual measurement value may be used as the position of the toolcenter point P9 with respect to the predetermined point P6.

Also, in the second embodiment as described above, the same effect as inthe first embodiment can be achieved.

Third Embodiment

Next, a third embodiment will be described.

FIG. 18 is a diagram illustrating a robot system according to a thirdembodiment. FIG. 19 and FIG. 20 are diagrams for explaining step S11.FIG. 21 is a diagram illustrating a captured image in step S14.

The third embodiment is similar to the calibration in the firstembodiment except that the calibration using the robot 1 in a statewhere the hand 17 (working unit) is worn is mainly performed. In thefollowing description, differences from the first embodiment describedabove will be mainly described, and description of similar matters willbe omitted.

As illustrated in FIG. 18, in the calibration of the third embodiment,the hand 17 for forming the marker 190 together with pressure sensitivepaper 81 (marker forming object) pressure sensitive paper 81 (markerforming object) is used. A specific configuration of the pressuresensitive paper 81 is not particularly limited, and general pressuresensitive paper can be used. The calibration in the third embodiment isthe same as that in the flow illustrated in FIG. 7 except that the hand17 and the pressure sensitive paper 81 are used.

Specifically, first, in step S11, the control unit 51 positions the hand17 in the imaging area S3 as illustrated in FIG. 19 and presses thepressure sensitive paper 81 with the hand 17 toward the work stand 71side. With this, as illustrated in FIG. 20, the marker 190 is formed onthe pressure sensitive paper 81 (step S11).

When it reaches the predetermined number of times (nine times in thethird embodiment) in step S14, the control unit 51 causes the imagingunit 3 to collectively image the nine markers 190, and stores the imagecoordinates of the respective markers 190 in the storing unit 52 (StepS14). The captured image 30 at this time is illustrated in FIG. 21.

As described above, in the third embodiment, the control device 5includes the external input and output unit 53 (I/F) having a functionas the “reception unit” that receives information (data) on the capturedimage 30 from the imaging unit 3 that can capture images and the controlunit 51 (processor) that controls driving of the robot 1 (which canexecute instructions relating to driving) including the robot arm 10having a function of performing work on the work target 91 and providedwith the hand 17 (marker forming unit) for forming the marker 190together with the pressure sensitive paper 81 (marker forming object).Further, the control unit 51 can execute correlation (calibration)between the robot coordinate system that is the coordinate systemrelating to the robot 1 and the image coordinate system that is thecoordinate system relating to the captured image 30. The control unit 51performs the correlation based on the plurality of robot coordinates(coordinates) of the predetermined point P6 (predetermined portion) ofthe robot arm 10 in the robot coordinate system when a plurality ofmarkers 190 are formed on the pressure sensitive paper 81 by the hand 17and the pressure sensitive paper 81 the plurality of image coordinates(coordinates) of the plurality of markers 190 in the image coordinatesystem.

According to such a control device 5, since it is possible to performcalibration using the robot 1 including the hand 17 and the pressuresensitive paper 81, vibration of the robot arm 10 and the like can bereduced as compared with that in the related art. For that reason, it ispossible to improve accuracy of the calibration. Since the replacementof the stamp 19 and the hand 17 with each other is unnecessary, thecalibration can be performed more quickly.

In particular, in the third embodiment, as described above, the markerforming object is configured by the pressure sensitive paper 81. Then,the control unit 51 controls driving of the robot 1 so as to form themarker 190 by bringing the hand 17 (marker forming unit) into contactwith the marker forming object configured by the pressure sensitivepaper 81.

With this, it is possible to easily form the marker 190 on the pressuresensitive paper 81 by bringing the hand 17 into contact with thepressure sensitive paper 81.

In the third embodiment, although the pressure sensitive paper 81 isused as the marker forming object, the marker forming object may beanything as long as it can form a marker together with the markerforming unit. For example, the marker forming object may be an objectcontaining magnet powder. In this case, the marker forming unit may be aworking unit or the like configured to contain a magnetic object, forexample, a metal material. The marker forming object may be a sheet orthe like configured to change its color according to applied pressure.

The “marker forming object” in the first embodiment described above maybe not the work stand 71 but the work stand 71 provided with thepressure sensitive paper 81. That is, the marker 190 may be formed onthe pressure sensitive paper 81 by the stamp 19.

Also, in the third embodiment described above, the same effect as in thefirst embodiment can be exhibited.

Another Configuration Example of Robot System

The robot system according to the third invention may be in the formsillustrated in FIGS. 22 and 23.

FIGS. 22 and 23 are block diagrams illustrating other examples of therobot system.

FIG. 22 illustrates the entire configuration diagram of a robot system100B in which a computer 63 is directly connected to the robot 1.Control of the robot 1 is executed directly by reading a command in amemory by a processor existing in the computer 63. The computer 63 hasthe function of the control device 5 described above. The computer 63may be built in the robot 1.

FIG. 23 illustrates the entire configuration diagram of a robot system100C in which the robot 1 with the controller 61 built therein and acomputer 66 are connected and the computer 66 is connected to a cloud 64via a network 65 such as a local area network (LAN). The control device5 described above can be configured by the controller 61, for example.However, control of the robot 1 may be executed by reading a command inthe memory by a processor existing in the computer 66, or executed byreading the command in the memory via the computer 66 by a processorexisting on the cloud 64. Accordingly, the control device 5 may beassumed as being configured by the controller 61 and the computer 66, ormay be assumed as being configured by the controller 61, the computer66, and the cloud 64.

Although the control device and the robot system according to theinvention have been described based on the illustrated embodiments, theinvention is not limited thereto, and the configuration of each unit canbe replaced with any configuration having the same function. Any otherconstituent elements may be added to the invention. Further, respectiveembodiments may be appropriately combined.

In the embodiments described above, although a so-called six-axisvertical articulated robot is exemplified as the robot included in therobot system according to the invention, the robot may be another robotsuch as a SCARA robot, for example. The robot is not limited to asingle-arm robot, but may be another robot such as a dual-arm robot, forexample. Accordingly, the number of movable units is not limited to one,and may be two or more. Although the number of arms of the robot armincluded in the movable unit is six in the embodiments described above,it may be 1 to 5, or 7 or more.

The entire disclosure of Japanese Patent Application No. 2017-166490,filed Aug. 31, 2017 is expressly incorporated by reference herein.

What is claimed is:
 1. A control device comprising: a processor that isconfigured to execute computer-executable instructions so as to controla robot, wherein the processor is configured to: receive information ona captured image from an imaging device capturing an image from anoperator, control driving of a robot including a robot arm on which astamp that forms a marker on a marker forming object and an end effectorthat performs work on a work target are allowed to be provided by beingreplaced by each other, perform correlation between a robot coordinatesystem that is a coordinate system relating to the robot and an imagecoordinate system that is a coordinate system relating to the capturedimage, and perform the correlation based on a plurality of coordinatesof a predetermined portion of the robot arm in the robot coordinatesystem and a plurality of coordinates of the plurality of markers in theimage coordinate system when the plurality of markers are formed on themarker forming object by the stamp, in a case where the end effector isnot provided on the robot arm.
 2. A control device comprising: aprocessor that is configured to execute computer-executable instructionsso as to control a robot, wherein the processor is configured to:receive information on a captured image from an imaging device capableof capturing an image from an operator, perform work on a work targetand control driving of a robot that includes a robot arm on which astamp that forms a marker together with a marker forming object isprovided, perform correlation between a robot coordinate system that isa coordinate system relating to the robot and an image coordinate systemthat is a coordinate system relating to the captured image, and performthe correlation based on a plurality of coordinates of a predeterminedportion of the robot arm in the robot coordinate system and a pluralityof coordinates of the plurality of markers in the image coordinatesystem when the plurality of markers are formed on the marker formingobject by the stamp and the marker forming object.
 3. The control deviceaccording to claim 1, wherein the processor is configured to form themarker by bringing the stamp into contact with the marker formingobject.
 4. The control device according to claim 2, wherein theprocessor is configured to form the marker by bringing the stamp intocontact with the marker forming object.
 5. The control device accordingto claim 1, wherein the processor is configured to control driving ofthe robot so as to form the marker by bringing the stamp.
 6. The controldevice according to claim 2, wherein the processor is configured tocontrol driving of the robot so as to form the marker by bringing thestamp into contact with the marker forming object configured by pressuresensitive paper.
 7. The control device according to claim 1, wherein theprocessor is configured to receive an output from a force detectiondevice provided in the robot arm, and detect contact between the markerforming object and the stamp based on the output from the forcedetection device.
 8. The control device according to claim 2, whereinthe processor is configured to receive an output from a force detectiondevice provided in the robot arm, and detect contact between the markerforming object and the stamp based on the output from the forcedetection device.
 9. The control device according to claim 1, whereinthe processor is configured to obtain a relative relationship betweenthe robot coordinate system and the image coordinate system based on aplurality of coordinates in the robot coordinate system and a pluralityof coordinates of the plurality of markers in the image coordinatesystem when the plurality of markers are formed and then, perform thecorrelation based on the relative relationship.
 10. The control deviceaccording to claim 2, wherein the processor is configured to obtain arelative relationship between the robot coordinate system and the imagecoordinate system based on a plurality of coordinates in the robotcoordinate system and a plurality of coordinates of the plurality ofmarkers in the image coordinate system when the plurality of markers areformed and then, perform the correlation based on the relativerelationship.
 11. The control device according to claim 9, wherein theprocessor is configured to perform the correlation using coordinates inthe robot coordinate system of which the number is larger than thenumber of the coordinates in the robot coordinate system used whenobtaining the relative relationship and coordinates in the imagecoordinate system of which the number is larger than the number of thecoordinates in the image coordinate system used when obtaining therelative relationship.
 12. The control device according to claim 10,wherein the processor is configured to perform the correlation usingcoordinates in the robot coordinate system of which the number is largerthan the number of the coordinates in the robot coordinate system usedwhen obtaining the relative relationship and coordinates in the imagecoordinate system of which the number is larger than the number of thecoordinates in the image coordinate system used when obtaining therelative relationship.
 13. The control device according to claim 1,wherein the processor is configured to control driving of the robot soas to form the marker by the stamp provided at a tip end portion of therobot arm.
 14. The control device according to claim 2, wherein theprocessor is configured to control driving of the robot so as to formthe marker by the stamp provided at a tip end portion of the robot arm.15. A robot system comprising: a robot; and a control device thatcomprises a processor that is configured to execute computer-executableinstructions so as to control the robot; wherein the processor isconfigured to: receive information on a captured image from an imagingdevice capturing an image from an operator, control driving of a robotincluding a robot arm on which a stamp that forms a marker on a markerforming object and an end effector that performs work on a work targetare allowed to be provided by being replaced by each other, performcorrelation between a robot coordinate system that is a coordinatesystem relating to the robot and an image coordinate system that is acoordinate system relating to the captured image, and perform thecorrelation based on a plurality of coordinates of a predeterminedportion of the robot arm in the robot coordinate system and a pluralityof coordinates of the plurality of markers in the image coordinatesystem when the plurality of markers are formed on the marker formingobject by the stamp, in a case where the end effector is not provided onthe robot arm.
 16. A robot system comprising: a robot; and a controldevice that comprises a processor that is configured to executecomputer-executable instructions so as to control the robot; wherein theprocessor is configured to: receive information on a captured image froman imaging device capable of capturing an image from an operator,perform work on a work target and control driving of a robot thatincludes a robot arm on which a stamp that forms a marker together witha marker forming object is provided, perform correlation between a robotcoordinate system that is a coordinate system relating to the robot andan image coordinate system that is a coordinate system relating to thecaptured image, and perform the correlation based on a plurality ofcoordinates of a predetermined portion of the robot arm in the robotcoordinate system and a plurality of coordinates of the plurality ofmarkers in the image coordinate system when the plurality of markers areformed on the marker forming object by the stamp and the marker formingobject.
 17. A robot system according to claim 15, wherein the processoris configured to form the marker by bringing the stamp into contact withthe marker forming object.
 18. A robot system according to claim 16,wherein the processor is configured to form the marker by bringing thestamp into contact with the marker forming object.